On Unique Decodability, McMillan’s Theorem and the Expected Length of Codes

In this paper we propose a revisitation of the topic of unique decodability and of some of the related fundamental theorems. It is widely believed that, for any discrete source X, every “uniquely decodable” block code satisfies E[l(X1, X2, · · · , Xn)] ≥ H(X1, X2, . . . , Xn), where X1, X2, . . . , Xn are the first n symbols of the source, E[l(X1, X2, · · · , Xn)] is the expected length of the code for those symbols and H(X1, X2, . . . , Xn) is their joint entropy. We show that, for certain sources with memory, the above inequality only holds if a limiting definition of “uniquely decodable code” is considered. In particular, the above inequality is usually assumed to hold for any “practical code” due to a debatable application of McMillan’s theorem to sources with memory. We thus propose a clarification of the topic, also providing extended versions of McMillan’s theorem and of the Sardinas Patterson test to be used for Markovian sources. This work terminates also with the following interesting remark: both McMillan’s original theorem and ours are equivalent to Shannon’s theorem on the capacity of noiseless channels.